\section{Introduction}
Recently there has been a new class of data-intensive applications due to the widespread usage of on-line services in Internet. These services include multimedia streaming, computer networks, financial services, etc. Most of these applications require to provide low-latency response time in a dynamic work load scenario. Many work has been done to realize streaming operators in various accelerators, examples are {\it Streaming processing engine}\cite{spe}, {\it Cells}\cite{streamcell} and {\it GPUs}\cite{streamgpu}.

MapReduce\cite{mapreduce08} is a programming paradigm proposed and implemented in Google for massive data parallel computing. Since its emergence, it has attacted a great deal of attentions both from the academia and industry. It abstracts away the complicated distributing programming issues for developers by providing two simple interfaces, {\it i.e.} map and reduce. Programmers do not have to worry about things like data partitioning, scheduling, resource allocation, inter-node communication, fault-tolerance and failure-recovery. 

Even though MapReduce has shown its ability and popularity for easy programming model and strong run time support; it aims for simplicity at the expense of generality and performance and thus is only suited for embarassingly parallel data processing. There has been a lot of efforts to expand the idea into other domains. Dryad\cite{dryad07} extends MapReduce to a multi-input, multi-output system that can execute a generalized directed acyclic workflow.  Dryad emphasizes on the specification of the fine control over the application dataflow graph. This graph explicitly reflects the dependency and enables the developers to work at a suitable level of abstraction for writing scalable distributed applications.
%The design of the graph generating rules and the implementation of a graph description language is similar to a streaming applications. 

In this project, we propose to support streaming application using Mapreduce, for reasons that:
\begin{itemize}
\item MapReduce has proven to be a good and easy programming model; 
\item The run-time system support design from MapReduce can be borrowed for a better platform. For example, the fault-tolence feature takes partial credit for the popularity of Mapreduce. In the meantime, researchers are starting to delve into fault-tolerance issues in streaming \cite{streamfault}. It is interesting to see if some of the ideas in Mapreduce can be applied to streaming applications as well.
\item The increasing data flow in current steaming applications calls for a scalable and efficient programming model \cite{SystemS} \cite{spade};
\item There is some similarities between a streaming process and Mapreduce-style jobs. A streaming application can often be expressed using a graph, in which vertices, as processing elements (PE), are connected with streams. This is quite similar to Dryad's case. What's more, for real-time requirement of streaming application, the queries are usually conducted in a continuous fashion whenever enough data stream objects have been collected, and the output is directly streamed to the next PE avoiding disk operation as much as possible. This is again similar to concatenated Mapreduce jobs. 
\end{itemize}

However, there exist fundamental gaps between a distributed stream processing system and the MapReduce. First of all, they originate from two different design goals. Mapreduce is first designed for massive-data processing, therefore latency comes of the lower priority. In streaming, users usually require low response time for a given input load. Secondly, Mapreduce utilizes distributed files systems to provide strong fault tolerance. Most of the data flow passes through disks; which is inappropriate for streaming programs. Thirdly, current Mapreduce jobs are considered to be static in the sense that the input size is known at prior, which leads to rigid number of mappers and reducers during the run-time. In a typical streaming model, Data Stream Management System (DSMS) must handle multiple continuous data streams and support long-running queries, producing answers in a continuous and timely way. What's more, streams are often bulky, time-varying and non-predictable. This means that a DSMS must monitor and balance the load among PEs and reallocate resources to them non-stoppingly. Though this shares similarity with MapReduce, the monitoring measures are different, eg., memory usage and queue length in contrast to task completion percentage; and load balancing actions are completely different. %Finally, the fault-tolerance in a DSMS is different from MapReduce, since recalculation is useless here.It would be nice to be able to change the working nodes according to the input pressure on the system.

The remainder of the paper is structured as follows. First, we introduce the current design of the Hadoop and the necessary changes in streaming Hadoop. We then describe the detailed implementation of the streaming Hadoop. Next we evaluate our work by comparing the performance to the original Hadoop, as well as how the configuration affect the execution of the stream Hadoop. Finally we conclude this project with future work and propose the possible next step.
